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_ ABSTRACT 

To help teachers assess the variety cf available 
instructional materials, a comparative, feint-by-point evaluation is 
recommended. Two distinct approaches have teen identified: content 
analysis, and instructional design. Ccntent analysis is useful when 
selecting materials in factual curriculum areas, such as science and 
mathematics. It involves the compariecn of the instructional 
materials against the syllabus or course objectives, and should be 
performed by a subject specialist. The instructicnal design approach 
concentrates on four aspects: (1) educaticnal objectives; (2) scope 
and sequence of materials; (3) method of instruction; and (4) 
evaluation of student learning. (A three-page form which can be used 
¢o rate curriculum materials is included, age well age instructions for 
its use and instructions for comparing different materials). (GDC) 
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Assessing the Worth of 
Instructional Materials 


by Cedric Croft, 
NZCER ° 


In recent years there has been a veritable explosion in 
the range of instructional materials produced for 
classroom use: reading laboratories, spelling 
laboratories, study skill schernes, structured maths 
apparatus, science programmes, audio-visual packages, 
not to mention material for overhead projectors and 
listening posts. During the same time improvements 
have been made in the staple of teaching —the 


textbook. Today’s texts are more carefully written, and 


cover a wider range of topics than previously. They are 
also designed to appeal more to children, and often 
incorporate principles intended to make them more 
effective instructional tools. 

There also seems to be more firms and companies 
with textbooks or other instructional aids to sell, and 
they compete vigorously for a share of the market, 
setting aside large sums for promotion and advertising. 
As a result, schools and teachers are often bombarded 
with publishers’ materials, frequently full of 
unsupported claims, about the effectiveness of this 
particular textbook or that particular learning laboratory. 

Faced with a number of possibilities, how cana 
teacher decide if a new textbook or learning package is 
likely to be more helpful in reaching his instructional 


goals than the materials he is using at present? And ho. 
can he decide between two rival textbooks or learning 
packages? One approach is to make a compa:ative, 
point by point evaluation, along the lines of the 
procedure suggested below. 


Methods of evaluating instructional materials 


The first major study of instructional materials was 
completed over 20 years ago, by Cronbach, and 
reflected the dominant position that textbooks then 
held as the primary medium of instruction. More 
recently Eash and his colleagues have identified two 
distinct approaches to the evaluation of instructional 
materials, one concentrating on content, the other on 
instructional design. 
Content analysis 


~ 


Content analysis involves comparing the instructional 
material with the requirements of either the natior al 
syllabus or the school scheme, in order to judge 
whether the material covered is appropriate to the 
guidelines laid down by these documents. Conte-: 
analysis is obviously very important when selecting 
instructional material intended to enable students to 
master the factual content of the curriculum, 
particularly in ‘knowledge’ subjects like maths and 
Science, and should only be done by someone 
completely familiar with the subject matter being 
assessed. It is probably not quite as appropriate fu: .he 
selection of instructional material for reading, writing, 
Speaking and listening, where the objective is to develop 
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generalized skills, rather than teach specific subject 
matter. 


Instructional design analysis 


The second approach to the evaluation of instructional 
material focuses on its design or arrangement, rather 
than its content. However, if the material under 
evaluation is in an area where knowledge is regarded as 
a major outcome of instruction, then an analysis of 
content should precede evaluation of the instructional 
design. 


Four aspects of instructional design 


Eash suggests that in evaluating instructional materials 
we concentrate on four aspects: (i) objectives, (ii) scope 
and sequence of materials, (iii) method of instruction 
and (iv) evaluation. For each of these aspects devise a 
set of questions and then rate the material ona 
numerical scale. It might be useful to devise an 
evaluation form, with space for details such as the title 
of the book, its cost, number required if adopted, class 
or class levels for which it is intended, and the names of 
the teachers doing the evaluation. Note that it is 
important, when comparing two textbooks or sets of 
instructional material, that the same teacher or team of 
teachers rate each item. This is to help ensure 
uniformity of assessment in what is, admittedly, a 
largely subjective exercise. 


Comparison of instructional materials 


Having completed your evaluation, the next step is to 
compare the new materials with the old or with other 
materials that are available and which seem to serve the 
same purposes. This can be done readily by means of a 
simple graph or chart. 


— an. , 
Figure 1 represents a’hypothetical profile of two | 


mathematics textbook series. Iri the opinion of the 
evaluators the books are of equal value as far as scope 
and sequence are concerned, but in all other respects 
Series 1 is superior. Thus a rational basis exists for 
preferring one series to the other. 


If You Use This Rating Procedure: 


1 Ensure that the content of the instructional material is 
relevant to your course objectives, if you consider the 
content to be important. 

2 Make sure that the questions you ask of the 
instructional material are valid in nature. If the ones 
used on the examples above are unsuitable, develop 
your own questions. 

3 Have 2-4 people undertake the assessment of the 
materials. This will help improve the reliability of the 
ratings. 

4 If members of the evaluation team disagree by more 
than two points on any one scale try to ascertain the 
reasons for these differences. 

5 Average the ratings and construct one profile to 
summarize individual assessments. 

6 Use the profile as a basis for deciding what actic> to 
take, for example, purchase and use without 
modification; purchase and modify; purchase anc 
arrange suitable training in use; don’t purchase. 
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Figure 1: A Profile of Two Maths Series 
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A Model Evaluation Form Page ! of 3. 


School Class or Form 

Instructional 

Material —Title Publisher 
Year Cost Number Required 


1 Objectives 

(i) What are the stated objectives? 

(ii) If no objectives are stated what do they appear to be? 

(iii) What abilities, skills or processes are stressed in the objectives? 
(iv) Rate the overall quality of the objectives by marking a numbered 


point on the scale below. 
Objectives Rating Scale 
Objectives are 
unknown or of Objectives 
. little use to : are clear 
teacher and relevant 


2 Scope and sequence of materials 

(i) What major areas of content are covered? 

(ii) Howis the subject matter organized? 

(iii) Is the sequence clear and logical? 

(iv) Is the sequence appropriate for the intended class or pupil level? 

(v) Does the material (both scope and sequence) fit into the planned 
unit of study without modification? If not, what modifications are 
needed and how much of the material will be unused? 

(vi) Rate the overall quality of the scope and sequence by marking a 
numbered point on the scales below. 
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A MODEL EVALUATION FORM Page 2 of 3 
Scope and Sequence Rating Scales 


Scope inade- Scope 
quate 1 2 3 4 5 6 7 satisfactory 


Sequence un- Material care- 
suitable for fully arranged, 
intended well inter- 
group 1 2 3 4 5 6 7 related 


3 Method of Instruction 

(i) Are any particular method(s) of teaching suggested? If ‘yes’ 
continue. 

(ii) What are the feature(s) of the suggested methods(s)? 

(iii) Do the method(s) stress the role of the teacher, the participation of 
the pupil, or both. 

_ (iv) Rate the overall quality of the suggested method(s) by marking a 

numbered point on the scale below. 


Method Rating Scale 
No teaching Possible teach- 
methods des- ing methods are 
cribed ! 2 3 4 5 6 7 ~~ fully described 


4 = Evaluation 

(i) Is any information given on how the materials were tested during 
development? 

(ii) What test materials for the pupils are included? 

(iii) Is information given on the reliability and validity of these tests? 

(iv) Do the test materials allow a teacher to infer how well students are 
progressing towards the objectives of the instructional materials? 

{v) Rate the overall quality of the evaluation information and testing 
suggestions by marking a numbered point on the scales below. 


A MODEL EVALUATION FORM Page 3 of 3 
Evaluation Rating Scales 
No information Full data on 
on development evaluation and 
or testing of classroom 
materials 1 2 3 4 5 6 7 trials given 


Contains well 

designed test or 

assessment 

No materials : materials cover- 
for assessment ing all sub-units 
given or sug- of material and 


gested 1 2 3 4 5 6 7 all objectives 


5 . Overall assessment 

(i) ts the material easy to use? 

(ii) How does it compare in cost with material already in use? 

(iii) How does it relate to the curriculum? 

liv) Assess the overall value of the material by marking a numbered 
point on the scale below. 


Material seems : Material well 

weak in instruc- designed. Easy — 
ional design. to use. Good 
Difficult to use value for money. 

effectively. Excellent cover- 

Expensive. Does age of subject. 

not relate well to Of high interest 


curriculum 1 2 3 4 5 6 7 to pupils 


Recommendation: 


Evaluated by: 


